Multi-agent Learning Environment Research Articles

To Interfere or Not To Interfere: Information Revelation and Price-Setting Incentives in a Multiagent Learning Environment

Demand uncertainty and seller competition are substantial challenges for online platforms. In “To Interfere or Not To Interfere: Information Revelation and Price-Setting Incentives in a Multiagent Learning Environment,” Birge, Chen, Keskin, and Ward analyze whether and how an online platform should offer demand information or price incentives to the sellers participating on the platform. The authors show that, when facing uncertain demand, the platform could be better off by doing nothing—that is, not providing any information or incentives to the sellers. They also develop a strategic reveal-and-incentivize policy for the platform to choose when to start sharing information and offering rewards to coordinate the sellers’ pricing. They prove that the strategic reveal-and-incentivize policy achieves near-optimal profit performance for the platform.

A Survey of Deep Reinforcement Learning Based on Multi-Particle Environments

This paper extensively explores the application of deep reinforcement learning within multi-agent environments, a domain marked by intricate challenges and burgeoning opportunities. Multi-agent environments inherently entail the presence of multiple interacting intelligent agents, thereby amplifying the complexity of decision-making and control. Traditional reinforcement learning methods, when confronted with these intricate settings, manifest limitations that have prompted a compelling shift towards the fusion of deep learning and reinforcement learning. The journey commences by elucidating the multifaceted challenges and intricacies pervasive in multi-agent environments. It systematically delineates the constraints of conventional reinforcement learning techniques, underscoring their inadequacies in effectively addressing the multifarious complexities presented by these environments. As the narrative unfolds, it places a spotlight on the transformative potential of deep learning in surmounting the formidable challenges associated with decision-making in multi-agent settings. A critical examination ensues, encompassing a thorough review of pioneering applications of methodologies such as value functions, policy gradients, and actor-critic approaches within the realm of multi-agent systems. Through an exhaustive exploration of existing literature, this article endeavors to unveil the vanguard developments and persisting challenges within the domain of deep reinforcement learning in multi-agent environments. Furthermore, this paper conducts a comparative analysis of the performance of two prominent algorithms, namely Q-learning and the Deep Q Network (DQN) algorithm. This analysis serves as a compass, elucidating the trajectory of development within the sphere of deep reinforcement learning. Ultimately, this paper acts as a compass, guiding future research and innovation in the dynamic and promising field of deep reinforcement learning within multi-agent environments.

To Interfere or Not To Interfere: Information Revelation and Price-Setting Incentives in a Multiagent Learning Environment

We consider a platform in which multiple sellers offer their products for sale over a time horizon of T periods. Each seller sets its own price. The platform collects a fraction of the sales revenue and provides price-setting incentives to the sellers to maximize its own revenue. The demand for each seller's product is a function of all sellers' prices and some customer features. Initially, neither the platform nor the sellers know the demand function, but they can learn about it through sales observations: each seller observes its own sales, whereas the platform observes all sellers' sales as well as the customer feature information. We measure the platform's performance by comparing its expected revenue with the full-information optimal revenue, and design policies that enable the platform to judiciously manage information revelation and price-setting incentives. Perhaps surprisingly, a simple do-nothing policy does not always exhibit poor revenue performance and can perform exceptionally well under certain conditions. With a more conservative policy that reveals information to make price-setting incentives more effective, the platform can always protect itself from large revenue losses caused by demand model uncertainty. We develop a strategic-reveal-and-incentivize policy that combines the benefits of the aforementioned policies and thereby achieves asymptotically optimal revenue performance as T grows large.

Multi-Agent Reinforcement Learning based on Value Distribution

Reinforcement learning in a multi-agent setting is very important for real-world applications, but it brings more challenges than those in a single-agent environment. In the multi-agent setting, the agent generally has a bias of overestimation on the value function. In our work, we pay attention to the issue of overestimation bias with continuous actions in the multi-agent learning environment. We propose a method to reduce this bias by adopting the distributional perspective on reinforcement learning. We combine it within the framework of off-policy learning Actor-Critic and propose a novel approach Multi-Agent Deep Distributional Deterministic Policy Gradient (MAD3PG). We empirically evaluate it in three competitive and cooperative multi-agent settings. Our results show that in a series of difficult motor tasks the agents trained by MAD3PG significantly outperforms existing benchmark.

Modeling and development of a multi-agent space for the secondary school

One of the guiding principles of the Fourth Industrial Revolution is the need for lifelong learning. This determines the growing role of intelligent educational systems to provide the necessary learning resources and services to users at any time and any place. This article presents the modeling and development of an intelligent multi-agent learning environment for the secondary school, developed by a team of the DeLC laboratory at the University of Plovdiv “Paisii Hilendarski”, Bulgaria. The learners are placed at the focus of the environment by personal assistants supporting work with the environment.

Deep Multi-Critic Network for accelerating Policy Learning in multi-agent environments

Humans live among other humans, not in isolation. Therefore, the ability to learn and behave in multi-agent environments is essential for any autonomous system that intends to interact with people. Due to the presence of multiple simultaneous learners in a multi-agent learning environment, the Markov assumption used for single-agent environments is not tenable, necessitating the development of new Policy Learning algorithms. Recent Actor–Critic algorithms proposed for multi-agent environments, such as Multi-Agent Deep Deterministic Policy Gradients and Counterfactual Multi-Agent Policy Gradients, find a way to use the same mathematical framework as single agent environments by augmenting the Critic with extra information. However, this extra information can slow down the learning process and afflict the Critic with Curse of Dimensionality. To combat this, we propose a novel Deep Neural Network configuration called Deep Multi-Critic Network. This architecture works by taking a weighted sum over the outputs of multiple critic networks of varying complexity and size. The configuration was tested on data collected from a real-world multi-agent environment. The results illustrate that by using Deep Multi-Critic Network, less data is needed to reach the same level of performance as when not using the configuration. This suggests that as the configuration learns faster from less data, then the Critic may be able to learn Q-values faster, accelerating Actor training as well.

SMIX(λ): Enhancing Centralized Value Functions for Cooperative Multi-Agent Reinforcement Learning

This work presents a sample efficient and effective value-based method, named SMIX(λ), for reinforcement learning in multi-agent environments (MARL) within the paradigm of centralized training with decentralized execution (CTDE), in which learning a stable and generalizable centralized value function (CVF) is crucial. To achieve this, our method carefully combines different elements, including 1) removing the unrealistic centralized greedy assumption during the learning phase, 2) using the λ-return to balance the trade-off between bias and variance and to deal with the environment's non-Markovian property, and 3) adopting an experience-replay style off-policy training. Interestingly, it is revealed that there exists inherent connection between SMIX(λ) and previous off-policy Q(λ) approach for single-agent learning. Experiments on the StarCraft Multi-Agent Challenge (SMAC) benchmark show that the proposed SMIX(λ) algorithm outperforms several state-of-the-art MARL methods by a large margin, and that it can be used as a general tool to improve the overall performance of a CTDE-type method by enhancing the evaluation quality of its CVF. We open-source our code at: https://github.com/chaovven/SMIX.

Emergence of Grounded Compositional Language in Multi-Agent Populations

By capturing statistical patterns in large corpora, machine learning has enabled significant advances in natural language processing, including in machine translation, question answering, and sentiment analysis. However, for agents to intelligently interact with humans, simply capturing the statistical patterns is insufficient. In this paper we investigate if, and how, grounded compositional language can emerge as a means to achieve goals in multi-agent populations. Towards this end, we propose a multi-agent learning environment and learning methods that bring about emergence of a basic compositional language. This language is represented as streams of abstract discrete symbols uttered by agents over time, but nonetheless has a coherent structure that possesses a defined vocabulary and syntax. We also observe emergence of non-verbal communication such as pointing and guiding when language communication is unavailable.

Examining the predictive relationship between personality and emotion traits and students’ agent-directed emotions: towards emotionally-adaptive agent-based learning environments

The current study examined the relationships between learners' ($$N = 123$$N=123) personality traits, the emotions they typically experience while studying (trait studying emotions), and the emotions they reported experiencing as a result of interacting with four pedagogical agents (agent-directed emotions) in MetaTutor, an advanced multi-agent learning environment. Overall, significant relationships between a subset of trait emotions (trait anger, trait anxiety) and personality traits (agreeableness, conscientiousness, and neuroticism) were found for four agent-directed emotions (enjoyment, pride, boredom, and neutral) though the relationships differed between pedagogical agents. These results demonstrate that some trait emotions and personality traits can be used to predict learners' emotions directed toward specific pedagogical agents (with different roles). Results provide suggestions for adapting pedagogical agents to support learners' (with certain characteristics; e.g., high in neuroticism or agreeableness) experience of adaptive emotions (e.g., enjoyment) and minimize their experience on non-adaptive emotions (e.g., boredom). Such an approach presents a scalable and easily implementable method for creating emotionally-adaptive, agent-based learning environments, and improving learner-pedagogical agent interactions in order to support learning.

Tracking Learning Based on Gaussian Regression for Multi-agent Systems in Continuous Space

摘要: 提高适应性、实现连续空间的泛化、降低维度是实现多智能体强化学习（Multi-agent reinforcement learning，MARL）在连续系统中应用的几个关键. 针对上述需求，本文提出连续多智能体系统（Multi-agent systems，MAS）环境下基于模型的智能体跟踪式学习机制和算法（MAS MBRL-CPT）.以学习智能体适应同伴策略为出发点，通过定义个体期望即时回报，将智能体对同伴策略的观测融入环境交互效果中，并运用随机逼近实现个体期望即时回报的在线学习.定义降维的Q函数，在降低学习空间维度的同时，建立MAS环境下智能体跟踪式学习的Markov决策过程（Markov decision process，MDP）.在运用高斯回归建立状态转移概率模型的基础上，实现泛化样本集Q值函数的在线动态规划求解.基于离散样本集Q函数运用高斯回归建立值函数和策略的泛化模型. MAS MBRL-CPT在连续空间Multi-cart-pole控制系统的仿真实验表明，算法能够使学习智能体在系统动力学模型和同伴策略未知的条件下，实现适应性协作策略的学习，具有学习效率高、泛化能力强等特点. 关键词: 连续状态空间 / 多智能体系统 / 基于模型的强化学习 / 高斯回归

Accelerating reinforcement learning through implicit imitation

Imitation can be viewed as a means of enhancing learning in multiagent environments. It augments an agent's ability to learn useful behaviors by making intelligent use of the knowledge implicit in ...

Accelerating Reinforcement Learning through Implicit Imitation

Imitation can be viewed as a means of enhancing learning in multiagent environments. It augments an agent's ability to learn useful behaviors by making intelligent use of the knowledge implicit in behaviors demonstrated by cooperative teachers or other more experienced agents. We propose and study a formal model of implicit imitation that can accelerate reinforcement learning dramatically in certain cases. Roughly, by observing a mentor, a reinforcement-learning agent can extract information about its own capabilities in, and the relative value of, unvisited parts of the state space. We study two specific instantiations of this model, one in which the learning agent and the mentor have identical abilities, and one designed to deal with agents and mentors with different action sets. We illustrate the benefits of implicit imitation by integrating it with prioritized sweeping, and demonstrating improved performance and convergence through observation of single and multiple mentors. Though we make some stringent assumptions regarding observability and possible interactions, we briefly comment on extensions of the model that relax these restricitions.